On the Foundations of Expected Expected Utility

Intelligent agents often need to assess user utility functions in order to make decisions on their behalf, or predict their behavior. When uncertainty exists over the precise nature of this utility function, one can model this uncertainty using a distribution over utility functions. This view lies at the core of games with incomplete information and, more recently, several proposals for incremental preference elicitation. In such cases, decisions (or predicted behavior) are based on computing the expected expected utility (EEU) of decisions with respect to the distribution over utility functions. Unfortunately, decisions made under EEU are sensitive to the precise representation of the utility function. We examine the conditions under which EEU provides for sensible decisions by appeal to the foundational axioms of decision theory. We also discuss the impact these conditions have on the enterprise of preference elicitation more broadly.